Stream-Order and Order-Statistics∗

When processing a data-stream in small space, how important is the order in which the data arrives? Are there problems that are unsolvable when the ordering is worst-case, that can typically be solved in practice? Does the role of ordering become less significant when we permit algorithms to make multiple passes over the data-stream if the stream is not re-ordered between each pass? If we consider the stream ordered by an adversary, what happens if we restrict the power of the adversary? We study these questions in the context of quantile estimation, one of the most studied problems in the data-stream model. Our specific results include an O(log n)-space, O(log log n)-pass algorithm for exact selection in a randomly ordered stream of n elements. This resolves an open question of Munro and Paterson [Theor. Comput. Sci., 23 (1980), pp. 315–323]. We then demonstrate an exponential separation between the random-order and adversarial-order models: using O(polylog n) space, exact selection requires Ω(log n/ log log n) passes in the adversarial-order model. This is established via a new bound on the communication complexity of a natural pointer-chasing style problem and, in contrast to previous results, applies to fully-general randomized algorithms. We also prove the first fully general lower-bounds in the random-order model: finding an n-approximate median in the single-pass random-order model with probability at least 9/10 requires Ω( √ n1−3δ/ log n) space. ∗Part of this work originally appeared in PODS 2006 [9] and ICALP 2007 [10]. †Dept. of Computer and Information Science, University of Pennsylvania. Email: [email protected]. Supported in part by an Alfred P. Sloan Research Fellowship and by NSF Awards CCF-0430376 and CCF-0644119. ‡Information Theory and Applications Center, University of California, San Diego. Email: [email protected]. Part of this work was done while the author was at the University of Pennsylvania.